Re-balance roles deterministically instead of randomly #209
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
carlcsaposs-canonical
requested review from
juditnovak,
phvalguima and
Mehdi-Bendriss
phvalguima
reviewed
Mar 26, 2024
(for auto-generated roles) Algorithm: - if odd # of nodes: all nodes are cluster manager eligible - if even # of nodes: node with highest unit number is data node, all other nodes are cluster manager eligible Replaces old algorithm of `random.choice` one of the nodes that should change roles This will enable in-place upgrades to be ordered from highest to lowest unit number even if roles are re-balanced (for HA [e.g. because of network cut]) while upgrade is in-progress That allows upgrade/rollback to be coordinated without leader unit setting information in peer databag, which allows partial rollback even if leader unit is in error state --- Also, this algorithm ensures that if a unit needs to be restarted for HA while upgrade in-progress, a unit (the highest number unit) on the newer version of the charm/workload will be upgraded. This reduces the likelihood that: - one of the last units on the older version restarts and - the cluster manager switches to operating with the new workload version (i.e. it stops operating in compatability mode with the old version—see https://discuss.elastic.co/t/rolling-upgrades-master-nodes-voting-config-exclusions/320463/2) which means that units on the old version cannot re-connect to the cluster
Highest unit number cannot always be determined (e.g. if unit hasn't joined peer relation). Unable to use `planned_units` since unit numbers are not necessarily sequential
max([]) raises ValueError
carlcsaposs-canonical
force-pushed
the
deterministic-role-re-balancing
branch
from
319a736 to
0b97602
Compare
phvalguima
approved these changes
Apr 5, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
(for auto-generated roles)
Algorithm:
Replaces old algorithm of
random.choiceone of the nodes that should change rolesThis will enable in-place upgrades to be ordered from highest to lowest unit number even if roles are re-balanced (for HA [e.g. because of network cut]) while upgrade is in-progress
That allows upgrade/rollback to be coordinated without leader unit setting information in peer databag, which allows partial rollback even if leader unit is in error state
Context:
https://chat.canonical.com/canonical/pl/zpqoyei54ffr5kwgwkbimizhwo
https://warthogs.atlassian.net/browse/DPE-3878
Also, this algorithm ensures that if a unit needs to be restarted for HA while upgrade in-progress, a unit (the highest number unit) on the newer version of the charm/workload will be upgraded. This reduces the likelihood that:
which would mean that units on the old version could not re-connect to the cluster